This invention relates generally to displaying and managing windows and images within a multiple display area environment where at least one of the display areas has a larger pixel size than at least one other of the display areas
A typical computer system includes a computer having a central processing unit, an input/output unit and memory containing various programs used by the computer such as an operating system and one or more application programs. An end-user of the computer system communicates with the computer by means of various input devices (keyboard, mouse, pen, touch screen, voice, etc.), which transfer information to the computer via the input/output unit. The computer responds to this input data, among other ways, by providing responsive output to the end-user, for example, by displaying appropriate text and images on the screen of a display monitor.
Operating systems often include a graphical user interface (“GUI”) by which the operating system and any applications it may be running (e.g., a word processing program) may communicate with the end-user. One commonly used GUI implementation employs a desktop metaphor in which the screen of the monitor is regarded as a virtual desktop. The desktop is an essentially two-dimensional working template area supporting various graphical objects, including one or more display regions. Information is displayed on the desktop within the display regions (e.g., window, dialog box, pop-up menu, pull-down menu, drop-down list, icon), which typically are rectangular in shape, although many shapes and sizes are possible. Each display region may be dedicated to a specific application or to the operating system under which the applications are running. By manipulating a cursor (such as with standard point & click techniques), an end-user can manage the display regions as desired, for example, by creating new display regions or eliminating old ones, or by resizing or repositioning the display regions to fit the end-user's needs. The end-user may “activate” a particular display region and its associated application, for example, by “clicking” the cursor when it appears within the desired region.
The screen size and resolution available to consumers has grown over the past years, but not as fast as the increase in storage and computational power has empowered users to work with larger data objects. For many tasks involving visual representations, the display thereby has become the bottleneck of computer systems. When a user's display is not able to display the number of pixels required for displaying all the desired information at once, users have the following choices:                (a) They can navigate (e.g. zoom and pan) the display manually to acquire the information sequentially. Additional navigation means additional user effort.        (b) They can replace the current display with a display able to display the required number of pixels, i.e. a “large high-resolution display”. Current technology is able to provide large high-resolution displays, but technologies proposed so far for such displays are still cost-intensive, space-intensive, or both, which has prevented these technologies from reaching the mass market.        (c) They can use an appropriate visualization technique that allows fitting the required data into a small screen by reducing the space allocated for irrelevant information. The two main approaches utilized in information visualization techniques are overview plus detail views (B. Shneiderman. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Third edition. Reading Mass.: Addison-Wesley, 1998.) and fisheye views (George Fumas, “Generalized Fisheye Views,” CHI '86 Proceedings, pp. 16-23).        
Overview plus detail visualizations use two distinct views: one showing a close-up and the other showing the entire document. The drawback of this approach is that it requires users to visually switch back and forth between the two distinct views and to reorient themselves every time they switch. Fisheye views avoid the distinction between two views by keeping adjacent information together. The switching between detail region and periphery is thereby accelerated. However, the downside of this approach is that it introduces distortion, which makes some content, for example photographic content, difficult to recognize. Both of these visualization techniques use different scaling for the different display regions, making it difficult to visually compare sizes and lengths between objects located in different regions.
To alleviate this problem, a computer system with a display called a “mixed resolution display” has been used. Mixed resolution displays combine two or more display units with different resolutions such that the geometry of displayed images is preserved. Objects displayed across multiple display units preserve size and shape, although their resolution changes.
There are two different ways of perceiving a mixed resolution display. Firstly, mixed resolution displays can be considered normal, monitor-sized displays that are enhanced with additional low-resolution display space in the periphery. Secondly, mixed resolution displays can be considered large low-resolution displays that are enhanced with a high-resolution region in the center, similar in concept to a “magic lens”. For a description of the “magic lens” system please see a paper by: Bier, E. A., Stone, M. C., Pier, K., Buxton, W., and DeRose, titled “T. D. Toolglass and magic lenses: the see-through interface” in the Proceedings of the 20th annual conference on Computer graphics, 1993, Pages 73-80.
A study by Jonathan Grudin (Grudin J., “Partitioning Digital Worlds: Focal and Peripheral Awareness in Multiple Monitor Use”, pages 458-465 of the Proceedings of the SIGCHI conference on Human factors in computing systems, CHI 2001, ACM Pressshows that users do not use a combination of two or more display units as a single display area, even though they show adjacent parts of the same computer desktop. The gap between the two display units may be accountable for this behavior. In order for an image displayed on a mixed resolution display to be perceived as a single image, the following basic properties of the image have to be preserved.                (a) Geometry-preservation: The geometry of displayed images should be distorted as little as possible. Angles, the ratio between lengths, and the ratio between surfaces of the displayed image should correspond as closely as possible to those of the image when it was created. Images that were created by projecting onto a flat surface (e.g. the film in a camera or the projection plane in a 3D rendering program) are best displayed by displaying using a flat surface, so that angles, distance relations, and size relations are preserved.        
Multiple monitor configurations have been used to create hybrid displays. For example see U.S. Pat. No. 6,018,340, titled “Robust Display Management in a Multiple Monitor Environment”, by Butler et al., issued on Jan. 25, 2000. However this implementation, does not offer the size preservation described above. When an image is displayed across two or more monitors that display their content using different pixel sizes (e.g. when the user is moving an image from one monitor to another), the two portions of the image on the individual monitors are displayed in different sizes, disrupting the user's perception of a continuous image.                (b) Color-continuity: The colors in the image should be retained, as closely as practicable to the colors in the image when it was created. For example, if two points in the image were the same color in the original image, then they should have the same or very similar colors in the displayed image.        (c) X/Y continuity: The gap between the visible display regions of the individual display units (the X/Y gap) should be as small as possible for a viewer located directly in front of the mixed resolution display.        (d) Z-continuity: Points that were in the same plane in the original image should be in the same plane or close to the same plane in the displayed image. The distance between the display planes of two or more display units (the Z gap) should therefore be as small as possible.        (e) Time-continuity: Dynamic images should preserve as closely as practicable lapsed time continuity between events. For example, in a computer animation, a video, etc. two changes that take place with a certain time distance in the original dynamic image should happen in the same order and with the same time distance in the displayed image.        
Multiple monitor configurations to extend the user's display space have not always been able to maintain this parameter. For example, in a paper by Feiner, S. and Shamash, A., titled “Hybrid user interfaces: breeding virtually bigger interfaces for physically smaller computers”, Proceedings of the Fourth Annual ACM Symposium on User Interface Software and Technology, pages 9-17, 1991, a system is described showing a hybrid display consisting of “goggles” worn by the user along with a single monitor. This solution requires tracking the user's head position. The lag resulting from the tracking mechanism inserts an undesirable time-discontinuity.
Mixed resolution displays try to address some of these criteria. Mixed resolution displays combine two or more display units with different resolutions such that basic properties of an image displayed on it are preserved. When images elements are displayed across multiple display units of a mixed resolution display, the image elements are displayed using the same size and shape, although they are displayed on display units with differently sized pixels. Additionally, the system introduces no inherent time lag between display units.